Parallel back propagation training algorithm for

| This paper describes several algorithms, mapping the back propagation learning algorithm onto a large 2-D torus architecture. To obtain high speedup, we have suggested an approach to combine the possible parallel aspects (training set parallelism, node parallelism and pipelining of training patterns) of the algorithm. Several algorithms were implemented on a 512 processor Fujitsu AP1000 and compared using NETtalk, a network that transforms text to speech. For a large number of processors we obtained best performance for the proposed combined algorithm, reaching 76 million weights updated per second (MCUPS) using 512 processors. Our results show that to obtain high speedup on a large number of processors, all types of parallelism in the back propagation algorithm ought to be considered to be combined. Also, highly parallel general purpose computers can compete with the performance of neurocom-puters.